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ABSTRACT 

The Operator Control Station of the J PL/NASA 
Telerobot Demonstration System provides an 
efficient man-machine interface for the 
performance of telerobot tasks. Its hardware 
and software have been designed with high 
flexibility. It provides a feedback-rich 
interactive environment in which the Operator 
performs teleoperation tasks, robotic tasks, and 
telerobotic tasks at ease. This paper discusses 
the to-date operational experiences of this 
system, particularly related to the 'Object 
Designate' Process and the Voice Input/Output 
Process. 

INTRODUCTION 

The Operator Control Station (OCS) of the 
JPL/NASA (Jet Propulsion Laboratory / National 
Aeronautics and Space Administration) 
Telerobot Demonstration System provides the 
man-machine interface between the Operator 
and the System. It provides all the hardware and 
software for accepting human input for the 
direct and indirect (supervised) manipulation of 
the robot arms and tools for task execution. 
Hardware and software are also provided for the 
display and feedback of information and control 
data for the Operator's consumption and 
interaction with the task being executed. It is a 
large system, complex yet designed for user- 
friendliness. 

Operational modes of the Telerobot Demonstrat- 
ion System include teleoperated control mode, 
autonomous robotic control mode, and 
telerobotic control mode which is a combination 
of the two former modes and takes the form of 
traded/shared control (also known as 
supervisory control). The execution of tasks in 


these various modes are all exercised from the 
Operator Control Station. 

The OCS contains state-of-the-art mechanical 
and computing hardware for providing control 
input to the System. It contains control 
software and human operator interface software 
for real-time and user-friendly interaction. 
Video displays for text, graphics and camera 
images are provided for operator consumption. 
Voice input/output is provided to reduce 
operator work-load. Data manipulation such as 
'Object Designate' capability is provided for 
efficient task definition and instantiation. 
Access to all Telerobot subsystems is provided 
for on-line control, monitoring, and off-line 
software development. 

This OCS has been installed and is operational in 
the JPL Telerobot Testbed since April 1989. As 
a stand-alone system, it has offered many 
rewarding and satisfying hours of operational 
experiences in the man-machine interface, 
particularly in the utility of the new "object 
designate" technology, and in the voice 
input/output process. This paper will discuss 
these two design elements and operational 
aspects in detail. 

OCS DESIGN REQUIREMENTS 

As part of the JPL Telerobot Demonstration 
System, the OCS acts as the center of action for 
the System, and necessarily interfaces with all 
the other subsystems. While referring all 
discussions on the overall System architecture 
to references [1,2,3], it suffices to say that OCS 
interfaces with the Task Planner & Reasoning 
Subsystem (TPR), Run-Time Control Subsystem 
(RTC), Manipulator Control and Mechanization 
Subsystem (MCM), and the Sensing & Perception 
Subsystem (S&P). 
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The OCS is configured for two operators, the 
Primary Operator and the Secondary Operator 
(also known as the Test Conductor). The primary 
station of the OCS has all the controls necessary 
for the Primary Operator to perform all 
functions independent of the secondary station. 
And the secondary station has a subset of the 
capabilities at the primary station, required for 
monitoring and secondary control purposes. The 
Operators' functions include the following: 

- System management functions: system 

startup/shutdown, setup, software configura- 
tion, other monitoring and diagnostics functions; 

- System operation functions: mode 

transitions, setting system parameters, system 
calibration, video switching, emergency halt 
(and other modes of halting), object data 
manipulation; 

- Manual teleoperation functions: hand 

motions for input to Hand controllers, setting 

subsystem parameters, establishing 
telepresence via visual, kinesthetic and 
proprioceptive feedback; 

- Autonomous robotic functions: to 

instantiate, monitor, supervise, direct, confirm 
and give permission to proceed all actions 
generated under autonomous planning and 
control; 

-Telerobotic functions: to execute hybrid 

commands of manual and autonomous control, 
and transitioning between various degrees of 
manual and autonomous controls. 

- Initiating and executing data logging 
functions for off-line analysis and system 
performance evaluation. 

Further details of the OCS requirements can be 
found in [4], Hardware and software design 
details are also documented in [5,6]. 


OCS HARDWARE/SOFTWARE CONFIGURATION 

Figure 1 shows the OCS, as implemented at the 
JPL Telerobot Testbed. The Primary Operator is 
shown working in conjunction with the 
Secondary Operator in front of their respective 
stations. Figure 2 shows the Primary Operator 
looking into the robot work site area. Direct 
viewing of the work site is available at the JPL 
Testbed; and it can also be screened off so that 
only indirect viewing is available. 

For the input of hand motions in manual 
telerobotic operations, the primary station 
houses the right and left Force Reflecting Hand 
Controllers and their electronics. Three 


overhead video monitors normally show the wing 
and overhead camera overviews of the task 
space. Two middle video monitors display 
special graphics required for the telerobot 
operations, eig. force-torque graphics from the 
robot arms and hands; simulated graphics for 
time-delay and predicted motions; etc. Directly 
in front of the Primary Operator are the stereo 
displays, necessary for the Operator to perform 
dexterous close-up operations. 

The primary station has three input media for 
command inputs into the OCS primary computer. 
Keyboard direct commands and system 
operations are provided for 'hands-off' 
operations, when the Operator is not using the 
hand controllers. A 4-tier menu input is 
provided for normal telerobot task operations. 
The same menu input commands are also 
duplicated by voice input, which is implemented 
with a flexible grammar and recognized as 
continuous speech commands. Iconic and 
hierarchical displays are also provided to the 
Primary Operator for the issuance of task level 
commands. 


The secondary station has a simpler design 
because of its secondary requirements. It houses 
two video monitors and the OCS secondary 
computer. While the Secondary Operator does 
not have any voice input capability, he can 
always enter all the commands via the OCS 
secondary computer terminal, via direct 
command inputs using the keyboard or via the 
menu selection process, displayed to him on the 
OCS secondary computer monitor. Here, the 
Secondary Operator cannot provide teleoperation 
hand controller inputs because of the lack of the 
input devices. All graphics, overlays and video 
images can be displayed to his two monitors, as 
routed by an OCS process of video switching. 

Common to both stations are the video switcher, 
which is configured to route any channels (16 
input channels of RGB color) to any combination 
of monitors or input devices (16 output 
channels). E.g. single camera views can be 
switched to desired monitors; multi-view can be 
fed and mixed in a video mixer; single-view 
video can be fed into a graphics machine for 
overlaying and mixing graphics on video images. 
For the "object designate" process, the latter 
mixing is done through special graphic overlay 
machines routed through the video switcher. 

Other audio mixing, amplification, video 
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encoding and decoding, recording equipments are 
also implemented in the OCS. A voice 
synthesizer is installed in conjunction with the 
voice recognizer. Voice annunciation is used for 
acknowledgement of command inputs and for the 
annunciation of certain messages, particularly 
critical messages. Additionally, both stations 
have their own individual emergency kill button, 
which can also be used for a special halt-retract 
function. Remote power on-off for the robot 
controllers are provided, and local power 
monitoring to the OCS racks are also installed. 

The OCS computer is configured by a SUN 3/160 
workstation as the primary computer, and with a 
SUN 3/60 workstation as the secondary 
computer. Both of them are connected to the 
ethernet for communication with the other 
subsystems of the Telerobot Demonstrator 
System. Network communication is considered 
more than adequate because of the low- 
bandwidth data rate between the OCS and the 
other subsystems. The multi-tasking and multi- 
view window capabilities on the Sun 
workstation are fully utilized for input and 
output purposes, as well as to provide terminal 
emulation communication to all Telerobot 
subsystems and users. 

OCS software consists of multiple processes 
performing the following functions: 

- Command Processing: OCS-specific 
commands and task level commands 

- Message Processing and Display 

- Ethernet and External Subsystem Interface 

- Video Switch Operation and Control 

- Wire Frame Object Designation 

- Voice Input/Output Processing 

While explanations on all these processes have 
been given in [6], the following discusses the 
last two processes in more detail. 

OBJECT DESIGNATION PROCESS 

This process permits the Operator to 
interactively update the position and orientation 
data of known objects by a mouse point-and- 
designate sequence. By so doing, any 
discrepancy, error, or unintentional displace- 
ment of objects - as represented in the initial 
data base - can by reconciled. 

Task situation 

Figure 3 shows the Telerobot Testbed robot 


work site. Two ORU (orbit replaceable unit) 
modules are shown sitting on the "stowbin", 
which is a raised and tilted rack with the 
appropriate SIC's (Standard Interface Connector, 
a standard Polar Platform design, which permits 
mechanical and electrical mating and coupling). 
The module to the right, which has a cubical 
shape, is better known as the electronics 
module; and the module to the left is the science 
instrument. These modules and the rack designs 
simulate existing designs. The SIC is a true 
copy of the current design (as of mid-1988). 

One of the telerobot tasks to be performed at 
the Telerobot Testbed is to remove the 
electronics module from a "baseplate" location, 
which is shown situated between the bases of 
the two manipulator robots. After removal from 
the baseplate location (single-arm as well as 
dual-arm), it will be inserted into the stowbin 
SIC adapter. 

This type of module removal, maneuver and 
insertion task scenario is to be performed by 
manual teleoperation, or by supervised telerobot 
control using compliant force-position control. 
For the latter, autonomous robot motion is 
commanded via 'macro' commands. And since 
autonomous motions are involved, 'accurate' data 
bases are necessary. The degree of accuracy 
required for the autonomous fine motions has 
been bounded at 5 millimeters, such that the 
chamfer and compliant control will permit the 
final correct seating. 

When the stowbin or rack does not have an 
absolute calibrated data base entry with the 
degree of accuracy just stated, the technique of 
'object designation' has to be invoked. 

The designation process 

The 'object designation' is initiated by having 
the following data sent to OCS: (a) object 

model; (b) object homogeneous transform; (c) 
camera models. 

The object model is a wireframe description, 
basically consisting of a list of vertices of the 
object, and a list of the edg ,s of the object. The 
list of vertices is accompanied by the positions 
of the vertices relative to a local coordinate 
system. The homogeneous transform of this 
local coordinate system origin defines the 
object's location and orientation, as currently 
stored in the data base. A bounding box can be 
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computed from the above data points, and the 
box will be used in subsequent viewing 
manipulations. Lastly, the geometric models of 
the cameras (five cameras in the current 
Testbed setup) are sent to the OCS; these models 
define the 2-dimensional coordinates on the 
camera focal plane when given the x-y-z 
location of a point in the work space. 

Now the OCS is ready for the actual designation 
process. Up to two camera views can be 
selected; the two views are constrained by the 
two units of graphic overlay hardware in this 
current installation. Upon each monitor (camera 
view) selection, the overlaid video scene is 
presented on the monitor. 

At the top of the monitor screen are the 
fourteen menu choices (to be discussed later). 
If indeed the wire frame of the object fits into 
the work space as seen by the camera's field of 
view, the wire frame model will appear on the 
screen. Otherwise, a message will appear 
saying that the object is out of view. In that 
case, the Operator can choose a 'conjure' action, 
forcing the presentation of the object model 
into the workspace. Figure 4a shows such a 
conjured view of the electronics module wire 
frame. The actual module is also visible on the 
right side of the video screen. Also notice the 
solid and dotted edges of the wire frame, 
providing a pseudo 3-dimensional perspective. 
This view is from a right wing camera. 

If the module is not in a good perspective or 
viewing angle, certain rotations can be done to 
this model. The menu item at the top of screen, 
including 'left', 'right', 'top', 'bottom', 'rotate 
front' can be activated to show different views 
of the model. 

At this point, a point selection sequence can be 
performed to associate vertices of the wire 
frame model to the vertices of the object as 
seen on the screen. A user-friendly clicking 
process has been designed into this OCS 
software. Figure 4b shows the state where 1 
vertex has been associated (notice the circle 
pair surrounding the right vertex on the top 
face), and a second vertex Is being designated 
(notice the 'rubber-band' cursor). 

After selection of a few points on an individual 
screen, the point-fitting action can be activated 
that will exercise a least squares algorithm to 
fit the selected vertices to the designated 
locations. A minimum of three points will 


provide translational and rotational fitting. Any 
less number of points will be fitted with a 
translational fit only. Figure 4c shows the 
fitted wire frame model after three points were 
selected. 

Usually, a single-camera-view fit will not 
provide a very good fit, because of the lack of 
'depth'. Figure 4d illustrates the point. This is a 
view from a left wing camera, looking at the 
same fitted wire frame of Figure 4c. Even 
though the fit looks good in the right camera 
view, the need for improvement in the left 
camera view is evident. Normally, an orthogonal 
view will serve the purpose well, as in this 
second view. Figure 4e shows the results on the 
left view when an extra point, i.e a total of four 
points, is selected and fitted. The fit looks 
perfect, and so does the fit viewed from the 
right camera, shown in Figure 4f. After the fit, 
the root-mean-square error is always displayed 
to the Operator, on screen and by voice. 

The object designation process can now be 
terminated by either activating 'complete' or 
'cancel'. In the former, the new object location 
and orientation will be returned to the data base 
for updating. 

Experiences have shown that root-mean-square 
errors of 1.5 pixels or less can easily be 
obtained consistently upon the fitting of six 
vertices. Six to eight vertices are normally 
more than enough to define an object. For 
shallow objects, it is not always possible to 
pick even that many vertices. This kind of pixel 
accuracy translates directly into location 
accuracy. Depending on the focal length and 
zoom of the camera, and on the accuracies of the 
camera models, accuracies up to 5 millimeters 
are attainable. For cases when such accuracies 
are not attained because of different sources of 
errors, another technique need to be invoked, 
namely 'relative update' technique [1], 

VOICE INPUT/OUTPUT PROCESSING 

The OCS has been installed with a continuous 
speech recognition system, the VERBEX Series 
5000 system (which has its own simple voice 
synthesis module), and a voice synthesis 
system, the DECtalk DTC01 system. A custom 
set of vocabulary and grammar has been 
designed and implemented with the VERBEX 
system, specially designed for the OCS 
telerobotic operations. Standard English 
sentences in the form of ASCII text strings are 
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sent to the DECtalk for voice annunciation; and 
where necessary, specific phonemic phrases are 
sent so as to produce more natural sounding 
voiced messages. 

Voice input 

The vocabulary and grammar set was developed 
according to the needs of the Operator during his 
operation of a typical telerobotic sequence. 
Essentially, this set duplicates all the keyboard 
or menu inputs to all the processes in the OCS 
software on the SUN 3/160 Primary 
Workstation. This set is written on the SUN 
workstation as a text file, called a grammar file 
or grammar-definition file . Using a VERBEX 
supplied software (for the SUN development 
environment), this text file is converted into a 

binary, machine readable recognizer iiifi. The 

latter is transferred (i.e. downloaded) to the 
resident RAM memory and/or data cartridge in 
the VERBEX unit. 

For the Operator to use the voice recognition 
system, his voice has to be trained and stored as 
templates on a voice file . The VERBEX supplies a 
user friendly development facilities for the 
Operator to first 'enroll' the new words in the 
vocabulary set, and then to train on possible 
combinations of phrases permissible by the 
grammar. Typically, a session of training for 
this OCS Telerobot vocabulary/grammar set is 
90 minutes. Experience has shown that three 
training sessions normally produce very reliable 
voice files, where recognition accuracy and 
rejection accuracy could be well above 95%. 

A set of 120 vocabulary words has been designed 
for the current OCS Telerobot operation 
scenarios. These words are single words as 
well as compound words, strung together as they 
are continuously spoken. Examples are: activate, 
cancel, clear, command_confirm, the_upper_ 
left^display, the_stereo_cameras. 

Simple grammars are designed into the use of 
these vocabulary words. They consist of single 
word commands, noun or noun phrases, and 
prepositions/connectors. Examples of the single 
word commands are: initiate, select, switch, 

set_video_switch_defaults, command_confirm, 
activate, move, verbex_report (an escape 
command). Examples of the noun or noun phrases 
are: display, camera, the_upper_right_display, 

the_left_wing_camera, object_designate_mode, 
tool_mode. Examples of the prepositions and 
connectors are: switch camera alpha display 


Charlie using channel 2. 

For the two reasons of (a) that all vocabulary 
words are not used in one single telerobot 
session or operation sequence, and (b) that 
recognition accuracy can further be improved, 
the set of vocabulary words are grouped into six 
groups. These six groups are also consistent 
with the modes of telerobot operation: 
o Host mode 
o Video switch mode 
o Object designate mode 
o Vision arm manipulate mode 
o Teleoperation mode 
o Telerobot mode. 

When one of these six modes are selected, a help 
file is also displayed on the OCS console beside 
the command and message windows. Figures 
5(a)-5(f) show the OCS monitor displayed with 
the voice_command help file on the left window, 
during the above stated six modes. The complete 
voice vocabulary and grammars are listed in the 
help windows. (All the possible combinations 
can be derived from the lists, but obviously not 
enumerated in the window.) Shown also in the 
figures are the command window, normal 
message window and critical message window. 
The menu selections are also highlighted. 

Voice output 

In terms of voice synthesis, no special effort is 
needed for custom designing the voices or 
designing the phrases. The DECtalk provides six 
default voices with default parameters such as 
rate and pitch; where such parameters can 
easily be changed in the text string. The text 
strings sent to DECtalk need not be customized, 
and can be written in plain English with normal 
abbreviations and even with certain acronyms. 
Normal phrasing, with commas and periods, 
usually produces intelligible human speech. 
Where necessary to put emphasis on certian 
words, or where the English pronunciation 
deviates from standard rules, the phrase or the 
specific word can be written in special 
phonemic forms. Such phonemic forms bypass 
the extensive-yet-still-limited dictionary look- 
up rules in DECtalk, and will faithfully produce 
the correct accents and pronunciation so desired 
by the Operator or dictated by the contents and 
context of text phrase. 

These text phrases, i.e. voice messages, are 
contained in a ’Message_Definition_File' which 
is a text file to the OCS software. Thus, the 
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definition of the phrases, including the specific 
pronunication of the phrases, could be changed 
at will by the Operator (i.e. user, not necessarily 
a programmer), without the burden of having to 
recompile the whole OCS software. 

Likewise, since the DECtafk requires as input 
text files written in ASCII strings, it can be 
used to annunciate phrases of any language. The 
only limitation is the roughly 40 phonemes, i.e. 
basic unit of sounds, of the English language. 
These phonemes contain all the English vowels, 
unstressed vowels, dipthongs, syllabic 
consonants, and consonants. 

SUMMARY 

The JPL/NASA Telerobot Demonstration System 
is scheduled for complete system integration 
and telerobot operation in the Fall of 1989. The 
Operator Control Station of the System has been 
developed, installed, operational, and integrated 
into the System infrastructure. While limited 
amount of exposure is gained to-date on 
complete telerobot task executions, the 
Operator Control Station has offered many 
rewarding and satisfying hours of operational 
experience as a stand-alone system. 

The man-machine interface has been shown to be 
effective, particularly in the utility of the new 
'object designation' technique. Through this 
technique, data bases can be effectively updated 
and the need for absolute data base calibration 
is greatly reduced. Future evolution of the same 
technique would also bring about interactive 
data base construction and better graphics 
interface to the Operator. The voice input and 


output system is not just a showy convenience, 
but is proven to be an indispensible companion 
to the Operator. Even though his workload is not 
being reduced by these technical advances, his 
operational efficiency is greatly enhanced and 
his hands are freed to do telerobot operations. 
Further experience with the Telerobot System 
operation and the man-machine interface 
operation in the upcoming months will certainly 
suggest improvements and additions to this 
Operator Control Station, 
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(a) HOST MODE 


(b) VIDEO SWITCH MODE 
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